Add more metrics to prometheus #5034

expenses · 2020-02-24T15:24:02Z

No description provided.

mxinden

Only give this a short review. Will take a more in-depth look soon.

client/tracing/src/lib.rs

mxinden · 2020-02-24T15:51:38Z

client/network/src/service.rs

+
+struct Metrics {
+	is_major_syncing: Gauge<U64>,
+	peers_count: Gauge<U64>,


Moving this to client/network looks a lot cleaner. 👍

mxinden

I think this pull request contains two not-closely related changes:

Pass a Prometheus metric registry down to client/network.
Introduce Prometheus as a tracing backend.

I would prefer this to be done in two distinct pull requests. In addition could you add a pull request description so that others can easily get up to speed when reviewing this?

(Please ignore my comments in case this was not yet ready for a review given the draft status.)

client/service/src/builder.rs

expenses · 2020-02-25T10:49:45Z

Agreed, I'll switch this PR to just be the networking tracing.

expenses · 2020-02-26T15:58:22Z

This is ready for review, but I'm happy to accept more ideas of metrics to track.

mxinden

Building a release binary right now to test this out. Code looks good to me.

Thanks for laying the ground work for other modules to be instrumented as well (e.g. authority-discovery).

mxinden · 2020-02-26T18:44:52Z

client/finality-grandpa/src/environment.rs

@@ -400,6 +419,17 @@ impl<B, E, Block: BlockT, N: NetworkT<Block>, RA, SC, VR> Environment<B, E, Bloc
 		self.voter_set_state.with(|voter_set_state| {
 			if let Some(set_state) = f(&voter_set_state)? {
 				*voter_set_state = set_state;
+
+				if let Some(metrics) = self.metrics.as_ref() {


Using environment.rs as a source of truth for the current round was a guess from my side. @andresilva could you approve that that is a good idea?

mxinden · 2020-02-26T18:47:25Z

Given its changes to Service I am guessing this needs a corresponding Polkadot pull request @expenses?

mxinden · 2020-02-26T20:47:39Z

✔️ Tested with https://github.com/paritytech/substrate/blob/master/.maintain/sentry-node/docker-compose.yml.

client/finality-grandpa/src/lib.rs

client/finality-grandpa/src/environment.rs

* adding unleash to ci (#5020) * adding unleash to ci * fixing formatting * with a dot please * alpha.3 now * do not publish testing helpers * remove old test-helpers cruft * fix cargo.lock * with alpha 4 * do not publish runtime-interface-test either * disable more test crates from publishing * switch to alpha.5 * replace tempdir with tempfile * update lru * switch to bytes 0.5 * release script fixes * switch on and to latest alpha * BUT THE SPACES * Fix: CI failing for some CLI tests (#5043) * Initial commit Forked at: 41bb219 Parent branch: origin/master * Increase killing grace period of CLI tests and display more info * Use --dev everywhere possible * Put pruning mode to its own params struct * Add pruning params to export-blocks command * Added missing file * Removed not-dev mode in tests * Add pruning mode to the revert command * Decrease killing grace period again * Move back unsafe_pruning to import_params * Applied proposed changes * aura: remove unused tx pool (#5046) * aura: remove unused transaction pool parameter * node-template: remove transaction pool from aura * aura: fix tests * Extend rust nightly detection in `wasm-builder` (#5021) Instead of just testing `cargo` and `rustup run nightly`, we now test the `CARGO` env variable and also scan non default nightlies. The user is also now able to select the toolchain with `WASM_BUILD_TOOLCHAIN`. * Add steps setting to benchmarking CLI (#5033) * Add steps setting to CLI, use max value to hit worst case. * Bump impl_version. * Apply review suggestion. * Remove indices from node-template (#5025) * Remove indices from node-template * Use identity lookup instead * Bump impl * clean cargo.toml * Fix documentation for "BlockBuilder::push_trusted" (#5051) * fix doc * rephrase * do not check unleash on every PR, only master and tags (#5054) * do not check unleash on every PR, only master and tags * move scripts folder * add signed-tag check to CI * remove publish-to-crates-io dependencies Co-authored-by: s3krit <pugh@s3kr.it> * prepare version to alpha.1 (#5055) bump version to -alpha.1 * Sync: validate block responses for required data (#5052) * Less verbose state-db logging * Validate block responses for block bodies * Update client/network/src/protocol.rs Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com> * Added validation test * Disconnect on missing header as well * Typo Co-Authored-By: André Silva <andre.beat@gmail.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: André Silva <andre.beat@gmail.com> * Make these chainspecs fields private (#5031) * Fix dockerfile (#5059) * Adds documentation for `wipe` and `commit` (#5053) * Adds documentation for `wipe` and `commit` This adds documentation to `wipe` and `commit` of `Externalities`. Besides that it removes the default implementation that would just panic and requires that all implementers of the trait implement the functions. * Update primitives/externalities/src/lib.rs Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com> Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com> * Fix the issue with `trybuild`'s `ui` tests (#4992) * repro ui bug * fix the tests * test with the new image * test without CARGO_HOME * test without fixes * test again * fix trybuild old versions * bump CArgo.lock * fix trybuild newest versions * bump Cargo.lock * trying on the latest image * bump Cargo.lock * run with the old image * ci will be green on the image from 2020-02-19 [skip ci] * bump Cargo.lock * Activate publishing of draft releases... (#5062) * Activate publishing of draft releases... ... And fix the message sending (missing parameter). * publish_draft_release.sh now checks latest... ... release on github rather than just a tag * Fix/div by zero (#5041) * Handle gas_price being zero separately * Bump spec_version * Add a unit & integration tests for gas price = 0 * set missing metadata fields, prepping alpha.2 (#5067) * setting first batch of descriptions * fix what I just broke * next batch * and pallets, too * last batch * set cargo.lock * keep'em dev-deps * bump version to alpha.2 * Fix revalidation not revalidating multiple times (#5065) * removes use of sc_client::Client from sc_finality_grandpa (#5030) * removes use of sc_client::Client from sc_finality_grandpa * code formatting * code formatting * removes use of sc_client::Client from sc_finality_grandpa * Remove deprecated host functions (#5038) Sadly we need to keep one function `ext_blake2_256`. This function is manually defined in `sp-core`. * removes use of sc_client::Client from sc_basic_authorship (#5050) * removes use of sc-client from sc-basic-authorship * refactor use of ProposerFactory * correct dep path * pallet-transaction-payment clean up (#5070) * Formatting clean up * Introduce separate setters for the fees. * *: Rename prometheus-exporter crate to substrate-prometheus-end… (#5076) This patch renames the crate for the following two reasons: 1. The prometheus-exporter crate introduces native in-process Prometheus style instrumentation to the Substrate project. Within the Prometheus ecosystem the term "exporter" is used for external processes exposing metrics for e.g. the Linux Kernel. In-process exposition would be described via the term "endpoint". 2. "prometheus-exporter" is generic and ignores the fact that it is only usable within the context of Substrate. In addition the name "prometheus-exporter" is already taken on crates.io. * rename `browser-utils` to `substrate-browser-utils` (#5079) * prepping for Alpha.3 (#5080) * Bump to alpha.3 * update gitlab-ci * Propagate DispatchError for benchmarks. (#5075) * Propagate DispatchError for benchmarks. * Apply review suggestions. * Use RuntimeString. * fix expect Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Add options to overwrite range bounds in benchmark command. (#5072) * Add --mins --maxs to benchmark command. * Apply review suggestions. * Update yamux to version 0.4.4. (#5086) * Remove more instances of futures01 (#4633) * Start removing last few instances of futures01 * Use to_poll on wasm * Revert "Use to_poll on wasm" This reverts commit 1c61728. * Fix fg test * Upgrade network test futures * Update offchain hyper version * Update service test * bump tokio to 0.2.10 * Removed some unneeded tokios * fixes * fix run_until_all_full * Make service test debuggable * Update client/offchain/src/api/http.rs Co-Authored-By: Demi Obenour <48690212+DemiMarie-parity@users.noreply.github.com> * Add service_test to test-int output * nitpicking * Finally fix test * Give up and revert client/serviec/test * Revert gitlab ci too Co-authored-by: Demi Obenour <demi@parity.io> * Make export blocks default to json on stdout (#5090) * Make export blocks default to json on stdout * Multiline instead of single line to stay under 100 cols * Change --json flag to --binary, defaulting to json * Offence reporting returns a result (#5082) * Offence reporting returns a result * Bump spec_version * Use unwrap instead of assertions * Fix more review grumbles * Update to libp2p 0.16.2 (#5088) * Remove request ID from the new protocol (#5049) * Make sure we remove a peer on disconnect in gossip (#5104) * Make sure we remove peers on disconnect in gossip state machine * Clear up the code * Add a comment * Expose `state-db` memory info (#5110) This exposes memory statistics from the state-db. * Change extrinsic_count to extrinsic_index in pallet-utility (#5044) Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com> * Add more metrics to prometheus (#5034) * Add a few things * Add finality_grandpa_round * fix fg tests * Nitpicks * Nitpicks * Fix name of prometheus crate * Update to SCALE 1.2.0 (#5113) This updates `parity-scale-codec` to `1.2.0`, which includes multiple performance improvements and a fix that bounds the capacity of a vector at decoding. * Lazy payouts (#4474) * TODOs * Remove superfluous: * partial implementation * full implementation * fix preferences * update comments * upgrade test WIP * fix more tests * fix cutoff * fix saturation * comment * upgrade mock * upgrade test * WIP migration * WIP migration * remove slot stake stuff * fix merge * migration of ledger * remove equalize from test * add test * fix * update doc * fix compilation * improve test readibility * improve doc * fix most todo * fix migration and test * remove println * WIP * add test and spec * weight * update doc * safer end_era * fix exposure of conversion * Revert "safer end_era" This reverts commit 72ff737. * fix useless put * exposure clipped * doc * fix payout with clipped * fix node runtime * add doc * pluggable and generalized staking module * remove print * update doc * refactor * improve documentation and implementation * fix test * Fix test * fix test * fix test * fix remove lowest stake from exposure, not biggest. * nomination index arguments in nominator_payout * add test * try to fix offence * apply slashed and bond eras until active era * doc * update spec version * add test upgrade from previous test environment * Apply suggestions from code review Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com> * nominators upgrade has been cleaned * dynamic history depth implementation * make current_era - history_depth included * Change equality check to start era to less than or equal * Use era specific validator prefs * Add print statement and comment about start era if < * fix next_reward overflow * make more check for bad era claim for zero cost * small refactor * code refactor + fix use of deprecated storage * fix wasm build * add comment * Fix tests * remove outdated comment * Apply suggestions from code review Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com> * gather active era information into one storage Co-authored-by: thiolliere <gui.thiolliere@gmail.com> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * impl on_runtime_upgrade Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com> Co-authored-by: Cecile Tonglet <cecile.tonglet@cecton.com> Co-authored-by: André Silva <andre.beat@gmail.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Marcio Diaz <marcio@parity.io> Co-authored-by: Nikolay Volf <nikvolf@gmail.com> Co-authored-by: s3krit <pugh@s3kr.it> Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com> Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com> Co-authored-by: Chevdor <chevdor@users.noreply.github.com> Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com> Co-authored-by: Denis Pisarev <denis.pisarev@parity.io> Co-authored-by: Eric <e.lim0322@gmail.com> Co-authored-by: Seun Lanlege <seunlanlege@gmail.com> Co-authored-by: Sergei Pepyakin <s.pepyakin@gmail.com> Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Ashley <ashley.ruglys@gmail.com> Co-authored-by: Toralf Wittner <tw@dtex.org> Co-authored-by: Demi Obenour <demi@parity.io> Co-authored-by: pscott <30843220+pscott@users.noreply.github.com> Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com> Co-authored-by: Gavin Wood <github@gavwood.com> Co-authored-by: thiolliere <gui.thiolliere@gmail.com>

* Add a few things * Add finality_grandpa_round * fix fg tests * Nitpicks * Nitpicks * Fix name of prometheus crate

* Initial idea of `on_runtime_upgrade` * Runtime storage for module version * Gui shawntabrizi runtime upgrade (#5118) * adding unleash to ci (#5020) * adding unleash to ci * fixing formatting * with a dot please * alpha.3 now * do not publish testing helpers * remove old test-helpers cruft * fix cargo.lock * with alpha 4 * do not publish runtime-interface-test either * disable more test crates from publishing * switch to alpha.5 * replace tempdir with tempfile * update lru * switch to bytes 0.5 * release script fixes * switch on and to latest alpha * BUT THE SPACES * Fix: CI failing for some CLI tests (#5043) * Initial commit Forked at: 41bb219 Parent branch: origin/master * Increase killing grace period of CLI tests and display more info * Use --dev everywhere possible * Put pruning mode to its own params struct * Add pruning params to export-blocks command * Added missing file * Removed not-dev mode in tests * Add pruning mode to the revert command * Decrease killing grace period again * Move back unsafe_pruning to import_params * Applied proposed changes * aura: remove unused tx pool (#5046) * aura: remove unused transaction pool parameter * node-template: remove transaction pool from aura * aura: fix tests * Extend rust nightly detection in `wasm-builder` (#5021) Instead of just testing `cargo` and `rustup run nightly`, we now test the `CARGO` env variable and also scan non default nightlies. The user is also now able to select the toolchain with `WASM_BUILD_TOOLCHAIN`. * Add steps setting to benchmarking CLI (#5033) * Add steps setting to CLI, use max value to hit worst case. * Bump impl_version. * Apply review suggestion. * Remove indices from node-template (#5025) * Remove indices from node-template * Use identity lookup instead * Bump impl * clean cargo.toml * Fix documentation for "BlockBuilder::push_trusted" (#5051) * fix doc * rephrase * do not check unleash on every PR, only master and tags (#5054) * do not check unleash on every PR, only master and tags * move scripts folder * add signed-tag check to CI * remove publish-to-crates-io dependencies Co-authored-by: s3krit <pugh@s3kr.it> * prepare version to alpha.1 (#5055) bump version to -alpha.1 * Sync: validate block responses for required data (#5052) * Less verbose state-db logging * Validate block responses for block bodies * Update client/network/src/protocol.rs Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com> * Added validation test * Disconnect on missing header as well * Typo Co-Authored-By: André Silva <andre.beat@gmail.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: André Silva <andre.beat@gmail.com> * Make these chainspecs fields private (#5031) * Fix dockerfile (#5059) * Adds documentation for `wipe` and `commit` (#5053) * Adds documentation for `wipe` and `commit` This adds documentation to `wipe` and `commit` of `Externalities`. Besides that it removes the default implementation that would just panic and requires that all implementers of the trait implement the functions. * Update primitives/externalities/src/lib.rs Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com> Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com> * Fix the issue with `trybuild`'s `ui` tests (#4992) * repro ui bug * fix the tests * test with the new image * test without CARGO_HOME * test without fixes * test again * fix trybuild old versions * bump CArgo.lock * fix trybuild newest versions * bump Cargo.lock * trying on the latest image * bump Cargo.lock * run with the old image * ci will be green on the image from 2020-02-19 [skip ci] * bump Cargo.lock * Activate publishing of draft releases... (#5062) * Activate publishing of draft releases... ... And fix the message sending (missing parameter). * publish_draft_release.sh now checks latest... ... release on github rather than just a tag * Fix/div by zero (#5041) * Handle gas_price being zero separately * Bump spec_version * Add a unit & integration tests for gas price = 0 * set missing metadata fields, prepping alpha.2 (#5067) * setting first batch of descriptions * fix what I just broke * next batch * and pallets, too * last batch * set cargo.lock * keep'em dev-deps * bump version to alpha.2 * Fix revalidation not revalidating multiple times (#5065) * removes use of sc_client::Client from sc_finality_grandpa (#5030) * removes use of sc_client::Client from sc_finality_grandpa * code formatting * code formatting * removes use of sc_client::Client from sc_finality_grandpa * Remove deprecated host functions (#5038) Sadly we need to keep one function `ext_blake2_256`. This function is manually defined in `sp-core`. * removes use of sc_client::Client from sc_basic_authorship (#5050) * removes use of sc-client from sc-basic-authorship * refactor use of ProposerFactory * correct dep path * pallet-transaction-payment clean up (#5070) * Formatting clean up * Introduce separate setters for the fees. * *: Rename prometheus-exporter crate to substrate-prometheus-end… (#5076) This patch renames the crate for the following two reasons: 1. The prometheus-exporter crate introduces native in-process Prometheus style instrumentation to the Substrate project. Within the Prometheus ecosystem the term "exporter" is used for external processes exposing metrics for e.g. the Linux Kernel. In-process exposition would be described via the term "endpoint". 2. "prometheus-exporter" is generic and ignores the fact that it is only usable within the context of Substrate. In addition the name "prometheus-exporter" is already taken on crates.io. * rename `browser-utils` to `substrate-browser-utils` (#5079) * prepping for Alpha.3 (#5080) * Bump to alpha.3 * update gitlab-ci * Propagate DispatchError for benchmarks. (#5075) * Propagate DispatchError for benchmarks. * Apply review suggestions. * Use RuntimeString. * fix expect Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Add options to overwrite range bounds in benchmark command. (#5072) * Add --mins --maxs to benchmark command. * Apply review suggestions. * Update yamux to version 0.4.4. (#5086) * Remove more instances of futures01 (#4633) * Start removing last few instances of futures01 * Use to_poll on wasm * Revert "Use to_poll on wasm" This reverts commit 1c61728. * Fix fg test * Upgrade network test futures * Update offchain hyper version * Update service test * bump tokio to 0.2.10 * Removed some unneeded tokios * fixes * fix run_until_all_full * Make service test debuggable * Update client/offchain/src/api/http.rs Co-Authored-By: Demi Obenour <48690212+DemiMarie-parity@users.noreply.github.com> * Add service_test to test-int output * nitpicking * Finally fix test * Give up and revert client/serviec/test * Revert gitlab ci too Co-authored-by: Demi Obenour <demi@parity.io> * Make export blocks default to json on stdout (#5090) * Make export blocks default to json on stdout * Multiline instead of single line to stay under 100 cols * Change --json flag to --binary, defaulting to json * Offence reporting returns a result (#5082) * Offence reporting returns a result * Bump spec_version * Use unwrap instead of assertions * Fix more review grumbles * Update to libp2p 0.16.2 (#5088) * Remove request ID from the new protocol (#5049) * Make sure we remove a peer on disconnect in gossip (#5104) * Make sure we remove peers on disconnect in gossip state machine * Clear up the code * Add a comment * Expose `state-db` memory info (#5110) This exposes memory statistics from the state-db. * Change extrinsic_count to extrinsic_index in pallet-utility (#5044) Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com> * Add more metrics to prometheus (#5034) * Add a few things * Add finality_grandpa_round * fix fg tests * Nitpicks * Nitpicks * Fix name of prometheus crate * Update to SCALE 1.2.0 (#5113) This updates `parity-scale-codec` to `1.2.0`, which includes multiple performance improvements and a fix that bounds the capacity of a vector at decoding. * Lazy payouts (#4474) * TODOs * Remove superfluous: * partial implementation * full implementation * fix preferences * update comments * upgrade test WIP * fix more tests * fix cutoff * fix saturation * comment * upgrade mock * upgrade test * WIP migration * WIP migration * remove slot stake stuff * fix merge * migration of ledger * remove equalize from test * add test * fix * update doc * fix compilation * improve test readibility * improve doc * fix most todo * fix migration and test * remove println * WIP * add test and spec * weight * update doc * safer end_era * fix exposure of conversion * Revert "safer end_era" This reverts commit 72ff737. * fix useless put * exposure clipped * doc * fix payout with clipped * fix node runtime * add doc * pluggable and generalized staking module * remove print * update doc * refactor * improve documentation and implementation * fix test * Fix test * fix test * fix test * fix remove lowest stake from exposure, not biggest. * nomination index arguments in nominator_payout * add test * try to fix offence * apply slashed and bond eras until active era * doc * update spec version * add test upgrade from previous test environment * Apply suggestions from code review Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com> * nominators upgrade has been cleaned * dynamic history depth implementation * make current_era - history_depth included * Change equality check to start era to less than or equal * Use era specific validator prefs * Add print statement and comment about start era if < * fix next_reward overflow * make more check for bad era claim for zero cost * small refactor * code refactor + fix use of deprecated storage * fix wasm build * add comment * Fix tests * remove outdated comment * Apply suggestions from code review Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com> * gather active era information into one storage Co-authored-by: thiolliere <gui.thiolliere@gmail.com> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * impl on_runtime_upgrade Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com> Co-authored-by: Cecile Tonglet <cecile.tonglet@cecton.com> Co-authored-by: André Silva <andre.beat@gmail.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Marcio Diaz <marcio@parity.io> Co-authored-by: Nikolay Volf <nikvolf@gmail.com> Co-authored-by: s3krit <pugh@s3kr.it> Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com> Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com> Co-authored-by: Chevdor <chevdor@users.noreply.github.com> Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com> Co-authored-by: Denis Pisarev <denis.pisarev@parity.io> Co-authored-by: Eric <e.lim0322@gmail.com> Co-authored-by: Seun Lanlege <seunlanlege@gmail.com> Co-authored-by: Sergei Pepyakin <s.pepyakin@gmail.com> Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Ashley <ashley.ruglys@gmail.com> Co-authored-by: Toralf Wittner <tw@dtex.org> Co-authored-by: Demi Obenour <demi@parity.io> Co-authored-by: pscott <30843220+pscott@users.noreply.github.com> Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com> Co-authored-by: Gavin Wood <github@gavwood.com> Co-authored-by: thiolliere <gui.thiolliere@gmail.com> * make compile * Add some tests * docs * Remove "useless" code * Fix merge and use n + 1 block number * Fix tests * unfix ui tests * Update on_initialize.stderr * fix test * Fix test * Bump spec * Remove `on_finalise` and `on_initialise` * Use bool for tracking runtime upgraded * typo * Support runtime upgrade with `set_storage` * Refactor migration code location * add trailing newlines * Remove old `IsUpgraded` flag * Update state root * Exhaustive match statement * Apply suggestions from code review Co-Authored-By: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com> Co-authored-by: Cecile Tonglet <cecile.tonglet@cecton.com> Co-authored-by: André Silva <andre.beat@gmail.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Marcio Diaz <marcio@parity.io> Co-authored-by: Nikolay Volf <nikvolf@gmail.com> Co-authored-by: s3krit <pugh@s3kr.it> Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com> Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com> Co-authored-by: Chevdor <chevdor@users.noreply.github.com> Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com> Co-authored-by: Denis Pisarev <denis.pisarev@parity.io> Co-authored-by: Eric <e.lim0322@gmail.com> Co-authored-by: Seun Lanlege <seunlanlege@gmail.com> Co-authored-by: Sergei Pepyakin <s.pepyakin@gmail.com> Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Ashley <ashley.ruglys@gmail.com> Co-authored-by: Toralf Wittner <tw@dtex.org> Co-authored-by: Demi Obenour <demi@parity.io> Co-authored-by: pscott <30843220+pscott@users.noreply.github.com> Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com> Co-authored-by: Gavin Wood <github@gavwood.com> Co-authored-by: thiolliere <gui.thiolliere@gmail.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Initial idea of `on_runtime_upgrade` * Runtime storage for module version * Gui shawntabrizi runtime upgrade (paritytech#5118) * adding unleash to ci (paritytech#5020) * adding unleash to ci * fixing formatting * with a dot please * alpha.3 now * do not publish testing helpers * remove old test-helpers cruft * fix cargo.lock * with alpha 4 * do not publish runtime-interface-test either * disable more test crates from publishing * switch to alpha.5 * replace tempdir with tempfile * update lru * switch to bytes 0.5 * release script fixes * switch on and to latest alpha * BUT THE SPACES * Fix: CI failing for some CLI tests (paritytech#5043) * Initial commit Forked at: 41bb219 Parent branch: origin/master * Increase killing grace period of CLI tests and display more info * Use --dev everywhere possible * Put pruning mode to its own params struct * Add pruning params to export-blocks command * Added missing file * Removed not-dev mode in tests * Add pruning mode to the revert command * Decrease killing grace period again * Move back unsafe_pruning to import_params * Applied proposed changes * aura: remove unused tx pool (paritytech#5046) * aura: remove unused transaction pool parameter * node-template: remove transaction pool from aura * aura: fix tests * Extend rust nightly detection in `wasm-builder` (paritytech#5021) Instead of just testing `cargo` and `rustup run nightly`, we now test the `CARGO` env variable and also scan non default nightlies. The user is also now able to select the toolchain with `WASM_BUILD_TOOLCHAIN`. * Add steps setting to benchmarking CLI (paritytech#5033) * Add steps setting to CLI, use max value to hit worst case. * Bump impl_version. * Apply review suggestion. * Remove indices from node-template (paritytech#5025) * Remove indices from node-template * Use identity lookup instead * Bump impl * clean cargo.toml * Fix documentation for "BlockBuilder::push_trusted" (paritytech#5051) * fix doc * rephrase * do not check unleash on every PR, only master and tags (paritytech#5054) * do not check unleash on every PR, only master and tags * move scripts folder * add signed-tag check to CI * remove publish-to-crates-io dependencies Co-authored-by: s3krit <pugh@s3kr.it> * prepare version to alpha.1 (paritytech#5055) bump version to -alpha.1 * Sync: validate block responses for required data (paritytech#5052) * Less verbose state-db logging * Validate block responses for block bodies * Update client/network/src/protocol.rs Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com> * Added validation test * Disconnect on missing header as well * Typo Co-Authored-By: André Silva <andre.beat@gmail.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: André Silva <andre.beat@gmail.com> * Make these chainspecs fields private (paritytech#5031) * Fix dockerfile (paritytech#5059) * Adds documentation for `wipe` and `commit` (paritytech#5053) * Adds documentation for `wipe` and `commit` This adds documentation to `wipe` and `commit` of `Externalities`. Besides that it removes the default implementation that would just panic and requires that all implementers of the trait implement the functions. * Update primitives/externalities/src/lib.rs Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com> Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com> * Fix the issue with `trybuild`'s `ui` tests (paritytech#4992) * repro ui bug * fix the tests * test with the new image * test without CARGO_HOME * test without fixes * test again * fix trybuild old versions * bump CArgo.lock * fix trybuild newest versions * bump Cargo.lock * trying on the latest image * bump Cargo.lock * run with the old image * ci will be green on the image from 2020-02-19 [skip ci] * bump Cargo.lock * Activate publishing of draft releases... (paritytech#5062) * Activate publishing of draft releases... ... And fix the message sending (missing parameter). * publish_draft_release.sh now checks latest... ... release on github rather than just a tag * Fix/div by zero (paritytech#5041) * Handle gas_price being zero separately * Bump spec_version * Add a unit & integration tests for gas price = 0 * set missing metadata fields, prepping alpha.2 (paritytech#5067) * setting first batch of descriptions * fix what I just broke * next batch * and pallets, too * last batch * set cargo.lock * keep'em dev-deps * bump version to alpha.2 * Fix revalidation not revalidating multiple times (paritytech#5065) * removes use of sc_client::Client from sc_finality_grandpa (paritytech#5030) * removes use of sc_client::Client from sc_finality_grandpa * code formatting * code formatting * removes use of sc_client::Client from sc_finality_grandpa * Remove deprecated host functions (paritytech#5038) Sadly we need to keep one function `ext_blake2_256`. This function is manually defined in `sp-core`. * removes use of sc_client::Client from sc_basic_authorship (paritytech#5050) * removes use of sc-client from sc-basic-authorship * refactor use of ProposerFactory * correct dep path * pallet-transaction-payment clean up (paritytech#5070) * Formatting clean up * Introduce separate setters for the fees. * *: Rename prometheus-exporter crate to substrate-prometheus-end… (paritytech#5076) This patch renames the crate for the following two reasons: 1. The prometheus-exporter crate introduces native in-process Prometheus style instrumentation to the Substrate project. Within the Prometheus ecosystem the term "exporter" is used for external processes exposing metrics for e.g. the Linux Kernel. In-process exposition would be described via the term "endpoint". 2. "prometheus-exporter" is generic and ignores the fact that it is only usable within the context of Substrate. In addition the name "prometheus-exporter" is already taken on crates.io. * rename `browser-utils` to `substrate-browser-utils` (paritytech#5079) * prepping for Alpha.3 (paritytech#5080) * Bump to alpha.3 * update gitlab-ci * Propagate DispatchError for benchmarks. (paritytech#5075) * Propagate DispatchError for benchmarks. * Apply review suggestions. * Use RuntimeString. * fix expect Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Add options to overwrite range bounds in benchmark command. (paritytech#5072) * Add --mins --maxs to benchmark command. * Apply review suggestions. * Update yamux to version 0.4.4. (paritytech#5086) * Remove more instances of futures01 (paritytech#4633) * Start removing last few instances of futures01 * Use to_poll on wasm * Revert "Use to_poll on wasm" This reverts commit 1c61728. * Fix fg test * Upgrade network test futures * Update offchain hyper version * Update service test * bump tokio to 0.2.10 * Removed some unneeded tokios * fixes * fix run_until_all_full * Make service test debuggable * Update client/offchain/src/api/http.rs Co-Authored-By: Demi Obenour <48690212+DemiMarie-parity@users.noreply.github.com> * Add service_test to test-int output * nitpicking * Finally fix test * Give up and revert client/serviec/test * Revert gitlab ci too Co-authored-by: Demi Obenour <demi@parity.io> * Make export blocks default to json on stdout (paritytech#5090) * Make export blocks default to json on stdout * Multiline instead of single line to stay under 100 cols * Change --json flag to --binary, defaulting to json * Offence reporting returns a result (paritytech#5082) * Offence reporting returns a result * Bump spec_version * Use unwrap instead of assertions * Fix more review grumbles * Update to libp2p 0.16.2 (paritytech#5088) * Remove request ID from the new protocol (paritytech#5049) * Make sure we remove a peer on disconnect in gossip (paritytech#5104) * Make sure we remove peers on disconnect in gossip state machine * Clear up the code * Add a comment * Expose `state-db` memory info (paritytech#5110) This exposes memory statistics from the state-db. * Change extrinsic_count to extrinsic_index in pallet-utility (paritytech#5044) Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com> * Add more metrics to prometheus (paritytech#5034) * Add a few things * Add finality_grandpa_round * fix fg tests * Nitpicks * Nitpicks * Fix name of prometheus crate * Update to SCALE 1.2.0 (paritytech#5113) This updates `parity-scale-codec` to `1.2.0`, which includes multiple performance improvements and a fix that bounds the capacity of a vector at decoding. * Lazy payouts (paritytech#4474) * TODOs * Remove superfluous: * partial implementation * full implementation * fix preferences * update comments * upgrade test WIP * fix more tests * fix cutoff * fix saturation * comment * upgrade mock * upgrade test * WIP migration * WIP migration * remove slot stake stuff * fix merge * migration of ledger * remove equalize from test * add test * fix * update doc * fix compilation * improve test readibility * improve doc * fix most todo * fix migration and test * remove println * WIP * add test and spec * weight * update doc * safer end_era * fix exposure of conversion * Revert "safer end_era" This reverts commit 72ff737. * fix useless put * exposure clipped * doc * fix payout with clipped * fix node runtime * add doc * pluggable and generalized staking module * remove print * update doc * refactor * improve documentation and implementation * fix test * Fix test * fix test * fix test * fix remove lowest stake from exposure, not biggest. * nomination index arguments in nominator_payout * add test * try to fix offence * apply slashed and bond eras until active era * doc * update spec version * add test upgrade from previous test environment * Apply suggestions from code review Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com> * nominators upgrade has been cleaned * dynamic history depth implementation * make current_era - history_depth included * Change equality check to start era to less than or equal * Use era specific validator prefs * Add print statement and comment about start era if < * fix next_reward overflow * make more check for bad era claim for zero cost * small refactor * code refactor + fix use of deprecated storage * fix wasm build * add comment * Fix tests * remove outdated comment * Apply suggestions from code review Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com> * gather active era information into one storage Co-authored-by: thiolliere <gui.thiolliere@gmail.com> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * impl on_runtime_upgrade Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com> Co-authored-by: Cecile Tonglet <cecile.tonglet@cecton.com> Co-authored-by: André Silva <andre.beat@gmail.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Marcio Diaz <marcio@parity.io> Co-authored-by: Nikolay Volf <nikvolf@gmail.com> Co-authored-by: s3krit <pugh@s3kr.it> Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com> Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com> Co-authored-by: Chevdor <chevdor@users.noreply.github.com> Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com> Co-authored-by: Denis Pisarev <denis.pisarev@parity.io> Co-authored-by: Eric <e.lim0322@gmail.com> Co-authored-by: Seun Lanlege <seunlanlege@gmail.com> Co-authored-by: Sergei Pepyakin <s.pepyakin@gmail.com> Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Ashley <ashley.ruglys@gmail.com> Co-authored-by: Toralf Wittner <tw@dtex.org> Co-authored-by: Demi Obenour <demi@parity.io> Co-authored-by: pscott <30843220+pscott@users.noreply.github.com> Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com> Co-authored-by: Gavin Wood <github@gavwood.com> Co-authored-by: thiolliere <gui.thiolliere@gmail.com> * make compile * Add some tests * docs * Remove "useless" code * Fix merge and use n + 1 block number * Fix tests * unfix ui tests * Update on_initialize.stderr * fix test * Fix test * Bump spec * Remove `on_finalise` and `on_initialise` * Use bool for tracking runtime upgraded * typo * Support runtime upgrade with `set_storage` * Refactor migration code location * add trailing newlines * Remove old `IsUpgraded` flag * Update state root * Exhaustive match statement * Apply suggestions from code review Co-Authored-By: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com> Co-authored-by: Cecile Tonglet <cecile.tonglet@cecton.com> Co-authored-by: André Silva <andre.beat@gmail.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Marcio Diaz <marcio@parity.io> Co-authored-by: Nikolay Volf <nikvolf@gmail.com> Co-authored-by: s3krit <pugh@s3kr.it> Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com> Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com> Co-authored-by: Chevdor <chevdor@users.noreply.github.com> Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com> Co-authored-by: Denis Pisarev <denis.pisarev@parity.io> Co-authored-by: Eric <e.lim0322@gmail.com> Co-authored-by: Seun Lanlege <seunlanlege@gmail.com> Co-authored-by: Sergei Pepyakin <s.pepyakin@gmail.com> Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Ashley <ashley.ruglys@gmail.com> Co-authored-by: Toralf Wittner <tw@dtex.org> Co-authored-by: Demi Obenour <demi@parity.io> Co-authored-by: pscott <30843220+pscott@users.noreply.github.com> Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com> Co-authored-by: Gavin Wood <github@gavwood.com> Co-authored-by: thiolliere <gui.thiolliere@gmail.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

expenses requested a review from mxinden February 24, 2020 15:24

expenses changed the title ~~Add more metrics to prometheus and add a tracing receiver.~~ Add more metrics to prometheus and a tracing receiver. Feb 24, 2020

mxinden reviewed Feb 24, 2020

View reviewed changes

mxinden reviewed Feb 25, 2020

View reviewed changes

client/service/src/builder.rs Outdated Show resolved Hide resolved

Add a few things

a522a58

expenses force-pushed the ashley-more-metrics branch from a3b0bd6 to a522a58 Compare February 25, 2020 11:01

expenses changed the title ~~Add more metrics to prometheus and a tracing receiver.~~ Add more metrics to prometheus Feb 25, 2020

expenses added 2 commits February 25, 2020 18:36

Add finality_grandpa_round

790ad91

fix fg tests

2062bb0

expenses marked this pull request as ready for review February 26, 2020 15:19

expenses requested review from andresilva and Demi-Marie as code owners February 26, 2020 15:19

expenses requested review from mxinden and mattrutherford February 26, 2020 15:19

Merge remote-tracking branch 'parity/master' into ashley-more-metrics

81fd6ae

mxinden approved these changes Feb 26, 2020

View reviewed changes

Merge remote-tracking branch 'parity/master' into ashley-more-metrics

5acd634

NikVolf reviewed Feb 27, 2020

View reviewed changes

client/finality-grandpa/src/lib.rs Outdated Show resolved Hide resolved

NikVolf reviewed Feb 27, 2020

View reviewed changes

client/finality-grandpa/src/environment.rs Outdated Show resolved Hide resolved

expenses mentioned this pull request Feb 27, 2020

Companion PR to Substrate#5034 (Add more metrics to prometheus) paritytech/polkadot#864

Closed

expenses added 2 commits February 27, 2020 16:31

Merge remote-tracking branch 'parity/master' into ashley-more-metrics

8c85687

Nitpicks

3b22f98

expenses requested review from NikVolf February 27, 2020 15:34

expenses added 2 commits February 27, 2020 17:11

Nitpicks

fc1ad16

Merge remote-tracking branch 'parity/master' into ashley-more-metrics

7aae893

Merge remote-tracking branch 'parity/master' into ashley-more-metrics

3845670

NikVolf approved these changes Feb 27, 2020

View reviewed changes

Fix name of prometheus crate

66f245a

expenses added this to the 2.0 milestone Mar 2, 2020

Merge remote-tracking branch 'parity/master' into ashley-more-metrics

ba296bd

mxinden mentioned this pull request Mar 2, 2020

Send to the telemetry how each connection is used #5087

Closed

expenses added B1-clientnoteworthy labels Mar 3, 2020

gnunicorn merged commit a201f34 into master Mar 3, 2020

gnunicorn deleted the ashley-more-metrics branch March 3, 2020 10:37

General-Beck pushed a commit to General-Beck/substrate that referenced this pull request Mar 4, 2020

Add more metrics to prometheus (paritytech#5034)

ea9eb5d

* Add a few things * Add finality_grandpa_round * fix fg tests * Nitpicks * Nitpicks * Fix name of prometheus crate

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add more metrics to prometheus #5034

Add more metrics to prometheus #5034

expenses commented Feb 24, 2020

mxinden left a comment

mxinden Feb 24, 2020 •

edited

Loading

mxinden left a comment

expenses commented Feb 25, 2020

expenses commented Feb 26, 2020

mxinden left a comment

mxinden Feb 26, 2020

mxinden commented Feb 26, 2020

mxinden commented Feb 26, 2020

Add more metrics to prometheus #5034

Add more metrics to prometheus #5034

Conversation

expenses commented Feb 24, 2020

mxinden left a comment

Choose a reason for hiding this comment

mxinden Feb 24, 2020 • edited Loading

Choose a reason for hiding this comment

mxinden left a comment

Choose a reason for hiding this comment

expenses commented Feb 25, 2020

expenses commented Feb 26, 2020

mxinden left a comment

Choose a reason for hiding this comment

mxinden Feb 26, 2020

Choose a reason for hiding this comment

mxinden commented Feb 26, 2020

mxinden commented Feb 26, 2020

mxinden Feb 24, 2020 •

edited

Loading